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ABSTRACT 

This report presents the key findings and 
recommendations of the Advisory Panel for the Head Start Evaluation 
Design Project. The principal audiences for the report are: (1) 
leadership and staff of the Administration for Children, Youth, and 
Families (ACYF) who have administrative responsibility for Head 
Start; and (2) local Head Start programs. Section 1 discusses eight 
recommendations for an overall strategy and general principles. 
Section 2 presents recommended research directions considered 
important for building a k'-owledge base. Section 3 offers the panel's 
recommendations for research support activities that are needed to 
create an infrastructure for Head Start research and evaluation. A 
variety of groups are encouraged to collaborate with ACYF by 
supporting Head Start studies or by launching parallel research and 
evaluation initiatives. Foundations, research institutes, 
universities, state and Ijcal education agencies, the Department of 
Education, local child development and early childhood education 
programs, and policymakers may find the report useful for framing 
research issues and approaches. (RH) 
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HEAD START RESEARCH AND EVALUATION: 
A Blueprint for the Future 



Recommendations of the Advisory Panel 
for the Head Start Evaluation Design Project 

INTRODUCTION 



Project Head Start is widely viewed as one of the nation's most successful social policy 
initiatives. It began with great optimism in 1965 as a summer child development program 
for childreL of low income families, and soon became a beacon of hope for these children 
and their families. Now 25 years old, Head Start has matured into a comprehensive and 
multifaceted full year intervention supporting and enhancing the capacities of children, 
families, and communities. 

With a national commitment to reach all eligible children. Head Start is poised for a major 
program expansion. This commitment is particularly salient in light of the national 
eiucational goal, established jointly by President George Bush and ine National 
Govcmors* Association, that all children shall enter school ready to leam. A strong and 
expanded Head Start is critical to meeting that goal and to advancing Americans twin aims 
of excellence and equity. 

Past research has demonstrated that quality programs for young children and their families 
can significantly improve their life course. Yet too litde is known about the extent to which 
and by what means Head Start maximizes the potential of different types of children and 
families, and how future public policy can chart a course toward that end. More specific 
research on children and families is needed to answer these questions. 

The Advisory Panel for the Head Start Evaluation Design Project 

This report presents the key findings and recommendations of the Advisory Panel for the 
Head Start Evaluation Design Project. The Panel was convened within the ^amework of a 
contract of Collins Management Consulting, Inc., with the Head Start Bureau of the 
Administration for Children, Youth, and Families (ACYF), Office of Human Development 
Services, Department of Health and Human Services. The Panel compnsea nationally 
recognized experts knowledgeable in one or more of the following areas: Head Start and 
other early intervention programs, research design, methodoiOgy, measurement, and policy 
analysis. The Panel members, who included representatives of various racial and ethnic 
groups, met as a group four times in the process of developing this report (December 1989, 
February 1990, April 1990, and June 1990). Selected Federal staff of the Department of 
Health and Human Services and the Department of Education also attended these mee^'ngs. 
A list of Panel members and their affiliations is included at the end of this report. 

Tl;c Panel presented the first opponunity in more than a decade and a half for a systematic 
analysis of research needs relevant to the %ture of Head Start. The group addressed past 
research and evaluation fii dings (including the evidence of the lasting effects of early 
intervention produced by research^oriented preschool prof,rams). Head Start's information 



needs, and ACYFs emerging strategy for expanding enrollment, upgrading quality, and 
introducing program changes. 

The Panel's initial charge was to recommend a series of options for the evaluation of the 
Head Start program. However, it soon became apparent that rather than recommending 
specific rnidies or design alternatives, the Panel should focus on defining an overall 
strategy and a set of guiding principles for the selection and conduct of future Head Start 
resc^h and evaluation efforts. The Panel strongly believes that to reach the nation's goals 
forHead Start, an integrated program of research and the establishment of a research 
infrastructure to support such efforts must be emphasized. This paper presents the Panel's 
consensus on a research blueprint for the future. 

Intended Audience 

Recognizing that there is widespread interest in the Head Start program and in other early 
intervention programs that focus on the needs of young children in low income families, 
the Panel's report is addressed to several audiences. The principal audience is ACYF 
leadership and staff who have administrative responsibility for Head Start, including 
research and evaluation, and local Head Start programs. However, a wide range of groups 
are encouraged to coUaborate with ACYF by supporting Head Start studies or by launching 
parallel research and evaluation initiatives. Accordingly, foundations, research institutes, 
universities, state and local education agencies, the Department of Education, local child 
development and early childhood education programs, and policymakers might find the 
Panel's report useful in framing research issues and approaches. 

The Panel's Recc/nmendations 

The Panel's recommendations are included in three sections of the report. First, the eight 
recommendations for an overall strategy and general principles are presented and 
discussed. These are followed by recommended research directions that are 
important for building a future knowledge base. The report concludes with the Panel's 
recommendations for research support activities that are needed to create an 
infirastrucnire for Head Start research and evaluation. 



OVERALL STRATEGY AND GiiNERAL PRINCIPLES 



Future Head Start research and evaluation efforts should be guided by a ci^^arly defined 
overall strategy and general principles. This section presents and discusses the eight 
principles formulated by the Panel 

1 . Head Start research and evaluation planning should be organized around 
two principal questions: 

• Which Head Start practices; maximize benefits for children and 
families with different characteristics under what types of 
circumstances? 

« How are gains sustained for children and families after the 
Head Start erperience? 

Research initiatives designed to answer these questions offer high promise of producing 
infonnation that will lead to continuing improvements in the quality of Head Start and other 
early childhood programs. Moreover, careful consideration of these pivotal questions will 
help ACYF identify Head Start factors and post-Head Start experiences that extend or 
attenuate die positive effects of the Head Start program. 

The nature of these two principal questions defines a new generation of research and 
evaluation which differs sifnificantly from past studies. First, they require much finer 
grained analyses, particularly with respect to independent variables. Second, diey require 
information specific to various subgroups of the Head Start population. Third, they 
recognize the contribution of the knowledge of context to the interpretation of data. Fourth, 
they acknowledge family functioning, both as a goal in itself and its importance in the 
mediation of the childs development. Hach of these research and evaluation implications 
will be discussed fimher in tiiis report. 

2 . An overall research strategy rather than a single large scale study is the 
appropriate framework for addressing critical Head Start research and 
evaluation questions. 

The Panel recommends strongly against a single large scale study of Head Start as the 
principal mechanism for seeldng answers to the pivotal research questions highlighted 
above. The methodological requirements for the new generation of research and evaluation 
issues do not lend themselves to large scale evaluations that treat Head Start as a single 
program. Head Start is not, in any simple sense, a uniform "treatment." The common 
denominator of Head Start programs nationwide is conformity to a set of regulatory 
performance standards that reflect comprehensive service requirements in education, parent 
involvement, social services, and health services. But programs are allowed and 
enco'!raged broad flexibility in how they deliver the required component services. 
Moreover, Head Start programs serve children in different regions and subeconomies of 
American society. They serve a number of minority groups and address issues of 
bilinguali^m and multiculturalism. They embody a variety of programmatic formulas and 
inventions created by local Head Start staff to respond to the unique needs of families in 
their communities. 
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An overaU strategy is needed to extend existing theory and state-of-the-an research methods 
and to provide a more comprehensive and in-depth knowledge base for improving the 
quality of services provided to children and families. The Panel proposes three key 
approaches to guide this new strategy. These are implementation of an integrated set of 
studies, use of diverse methodologies and identification of marker variables. 

Implementation of an Integrated and Coordinated Set of Research and 
Evaluation Studies Collectively Designed to Address the Major Questions. 

An advantage of multiple studies, as opposed to a single large project, is the capacity for a 
cluster of studies to complement one another at a single point in time and to build upon one 
another m uicrcmental stages over time. A further advantage is the ability to cross-validate 
findings using different methodologies and to test the hypotheses with different subgrouDs 
of programs and participants. ^ ^ 

pe research a.id evaluation studies should be designed to yield results that are interpretable 
tor specific subgroups of children and/or families and for specific localities These 
tindings, taken as a whole, could address the major questions of interest. In addition, the 
studies should be designed and conducted by a consortium of investigators who would 
contnbute a number of different perspectives and areas of expertise to ±e effort. Such an 
anangeinent permits much more control over the quality of the data than is possible in large 
scale studies that are under the direction of an individual invesdgator or contracting firm. 

The necessity for an integrated and coordinated set of studies cannot be overemphasized 
Although a large number of studies of Head Start were conducted over the past 25 years 
(panirularly m the first dozen years of the program's existence), what exists is a 
tracacnated accumuladon of studies that do not build upon one another. These efforts have 
yielded relatively litde in the way of an organized body of knowledge. Head Start research 
and evaluation, m general, has not been based upon well formulated program and oolicv 
questions. f j 

Use Df Diverse Methodologies 

The reed for different research designs depends on the state of knowledge and the 
partic' liar issues beuig explored. The proposed strategy draws upon diverse methodologies 
uvcluomg case, ethnographic, correlational, quasi-experimental, and experimental studies 
Ihe common denominator of these various designs should be an overall conceptual 
trame^vork guided by the principal questions. 

Correlational and quasi-experimental studies could be used to test hypothetical causal 
model J, models of "What works best for whom?" and of factors conducive to the 
maint<.nance of gams. Such studies could also identify variables that suggest causal 
influences or active ingredients." Experiments could be designed to test these hypotheses 
through treatment manipulations or modifications, or additions to existing Head Start 
programs m randomized trials. This is one example of how a particular issue could be 
pursued through several stages of inquiry. There are many variations on this theme 
depending on the particular questions to be explored and the available research and 
evaluaaon resources. 

It is imp«3rtant to note that, in the view o^ the Panel, randomized studies designed to 
comp;ire die effects of Head Start against the effects of nonparticipation ("treatment vs no 
treatment ) are generally no .onger viable options. First, as ACYF progresses toward the 



Administration's goal of universal services for all eligible children, the ootential for 
withholding services to form a control group, already difficult for ethical and practical 
reasons, will cease to be an option. Second, in view of the expansion of state and public 
school preschool programs and developmental child care, it is unrealistic to expect to find 
in most communities a representative group of "untreated" eligible children, even if Head 
Start services are not provided. 

There are a variety of ways to respond to these and other constraints on experimental 
options posed by changing societal realities. One approach would be to use random 
assignment at the level of individual Head Start programs, centers, classrooms, or groups 
of home visitors to test various experimental "add ons." The control comparison would be 
an individual Head Start program (or unit or units) from the same subpopulation that did 
not receive the "add ons." 

Case and ethnographic studies have played an increasingly important role in social science 
methodology. First, they are particularly helpful for hypothesis formulation. Second, 
knowledge of how programs actually operate, why something works or does not work, 
why some strategies work for children and families with a particular set of characteristics 
and not for others and the conditions under which they work are best informed by 
qualitative methodologies. Quantitative research does not replace qualitative kr.owledge, 
but instead builds upon it and requires it for valid interpretation. Both quantitative and 
qualitative research must conform to appropriate scientific criteria for assembling and 
analyzing evidence in order to insure the validity, reliability, and replicability of the 
findings. 

In the past, one of the detriments to the use of qualitative methods has been the high cost of 
the methodology. However, if high quality research is to be conducted to respond to the 
^ritical issues in Head Start, then future Head Start studies must use both qualitative and 
quantitative methods in a complementary fashion. 

Identification of Marker Variables 

Another necessary condition for an integrated and coordinated set of studies is the 
identification of a set of marker variables for child functioning, family functioning, 
program characteristics, and community characteristics. These core variables will make it 
possible to tie separate studies together. Identification of marker variables should take 
place early in the implementation of the next series of Head Start studies, either as an 
integral part of the initial studies or as a separate activity. Additional variables would be 
added to particular studies, depending on the specific questions under investigation. 
Marker variables should be sensitive to racial, ethnic, linguistic, and cultural diversity 
among Head Start subpopulations. 

In summary, the Panel recommends a strategy that involves an integrated set of small scale 
smdies involving different methodologies, different subgroups of the Head Start population 
and multiple investigators However, use of small scale studies does not mean that 
findings and policy decisions will be based on small samples. Nor does this mean that 
these studies would be simple in design. Rather, the recommended strategy will facilitate 
the convergence of results from multiple sources that will collectively incorporate robust 
findings from sophisticated designs. Thus the study outcomes will say something 
meaningful about the impact on subgroups of program participants. 




-6- 



1^ 



3. The diversity of Head Start children and families as well as the diversity 
of the communitie? in which they reside must be recognized explicitly in 
future evaluation md research. h j 

Head Start families and children are not all alike. The problems faced by an African- 
Amencan child growmg up m an inner city neighborhood ate only broadly like those of a 
Native American child on an Indian reservation, a child growing up in a depressed area of 
Appalachia, or a Spanish-speaking child in the migrant stream. It is essential that future 
research and evaluation on the efficacy of Head Start address the program effects for these 
diverse populations. 

fpedfic populations can be defined by such factors as presence and type of child 
disabilities, children s health status (including lingering effects of parental substance 
abuse), family composition and functioning, racial and ethnic status, linguistic a fferences 
geographic area of residence, and other variables (or combinations of variables) that 
encompass the wide diversity of such subgroups. 

The community attributes that may be relevant to program and policy development include 
the charactensacs of neighborhoods in which the children reside (for example, inner city 
rural, suburban, or migrant; areas of concentrated poverty and social dislocation; areas with 
imnunal health, education, and social services; and degree of homogeneity^eterogeneitv in 
the population). o j b j 

Previous research on early childhood programs has to some degree taken subpopulations 
into account by further subdivision of their original sample at the time of analysis 
However, the important subpopulation issues have not been addressed. Subpopulation 
issues must be considered and incorporated into the initial design of all future studies. 

The fiindamertal reason for considering subgroup populations from the outset of research 
pl^uig IS the likelihood that modifications in site selection, in the selection of independent 
and dependent variables, and in instrumentation may prove necessary to take into account 
specific subgroups. If a proper foundation is not laid at the research design phase, it will 
be difficult, and sometimes impossible, to rectify the situation during analysis. 

For example, readiness to learn is a reasonable objective for Head Start children. To ittain 
Head Stan s goal of social competence, program staff must facilitate the child's learning in 
developmentally appropnare ways and enhance future prospects for success in 'ormal 
schooling. However, this objective may need to be redefined, sometimes radically, for 
children with certain disabling conditions or for children whose dominant language is other 
Jian English. Each subgroup with disabiliries or with language differences can benefit 
from pmicipation m Head Start. However, the expected outcomes (dependent variables) 
may differ, with implications for measurement selection and data collection techniques. 
Such subgroup considerations are particularly salient in addressing the question "What 
works best for whom?'' 

4. Evaluation research must explicitly address diverse outcome indices 
related to children, families, communities and institutions. 

Historically, Head Start ha; embraced multiple goals affecting children, families, and other 
institutions and conditions within the local community. Yet in the past, studies of Head 
btart have overwhelmingly focused on ct.Ild outcomes, particularly outcomes in the 
cognitive and language domains. Little attention has been given to child outcomes in the 
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domains of socioemotional and physical development. Even less emphasis has been 
devoted to the impact on families, other institutions, and the community ecology. 



Child ortcomes should remain a central thrust of Head Start studies. However, the focus of 
future Head Start research and evaluation efforts should be widened to include a broader 
array of child outcomes and to encompass outcomes for families, communities, and 
institutions. 

Head Start is a two generation program that, in addition to the social competence goals for 
children, addresses goals for parents and other family members as priority outcomes in 
their own right. For example. Head Start's initiatives in the areas of family literacy, job 
training, and family support incorporate objectives of parental educational attainment, 
employability, and family self-sufficiency. ACYFs research agenda should give high 
priority to issues of family functioning, parent involvement in their children's development, 
family support, and family variables as mediating influences on child functioning. 

Examples of salient family outcomes of a Head Start experience include: 

• improved parenting skills; 

• increased parent-child interaction to promote child development and learning 
and to strengthen the family system; 

• improved expectations for children's future success in school; 

• increased parent involvement in schools and oLher community institutions; 

• reduced risk factors associated with family stress, including family 
violence, child abuse and neglect and substance abuse; 

• reduced dependence on welfare and heightened parental skills related to 
economic self-sufficiency (for example, improved literacy, adult education, 
and employability); and 

• improved access to and utilization of community family support services. 

In pursuit of its multiple program goals. Head Start has forged new relationships with 
agencies at the Federal, state and local levels and served as a catalyst in developing 
partnerships imong human services agencies. The nature of such collaboration is an area 
of inquiry in terms of the impact of Head Start's efforts on policy and on the delivery of 
social programs and must be taken into account in research design and in the selection of 
independent variables. 

5 . Multiple indicators and methods should be employed in the measurement 
of important outcomes. 

Evaluations should utilize, to the fullest extent possible, outcome indicators that 3xe 
reflective of the generally understood goals of Head Start and that convey the program's 
important effects. These outcome measures should be readily understandable to parents, 
program staff, policymakers, and the general public. In addition to carefullv selected 
standardized tests, outcome measures should include such straightforward indicators as 



ERLC 



-8- 
7^ 



whether or not a child is at grade level, is placed in a special class, or does or does not have 
an undetected (or identified but untreated) medical condition. 

Researchers should be mindful, however, that there are often complex dimensions of 
simple measures of the type recommended. For example, whether a child is retained in 
grade in the public schools is a relatively simple indicator. Why the child is retained in 
grade, the cntena by which this decision is made, and the implications for die child's future 
success in school may vary greaUy witiiin and across different school systems. Similarly, 
whether or not a child is in a special education class or experiencing "pull out" instmction is 
relatively easy to establish. What this means in terms of the child's placement in an 
appropriate educational environment that will best meet the child's learning and 
develq)mental needs is more con- )Iex. 

In general, previous Head Start studies used norm-referenced tests eitiier of I.Q. or 
achievement as die single metiiod for measuring program outcomes. There has been weU 
justiiied criticism of the use of I.Q. tests, panicularly for minority children. However 
there arc other norm-referenced tests tiiat are useful in detentnining die chUd's acquisition of 
csitain skills. In fact, familiarity widi "taking tests" and die particular skills involved in the 
tesi process can serve die child well in terms of future academic demands. 

Norm-referenced tests should be only one of several niethods for measuring child status 
Indicators of a child's status, such as reading ability, number competence, or ability to 
function as a confident social participant in classroom and school processes arc best arrived 
at through multiple methods, including observations, and ratings by teachers, parents, and 
peers. 

Care should be taken in die selection of child outcome instruments to keep in mind diat the 
ulmnate purpose iS to assess program effectiveness. Widi respect to school readiness and 
achievement, for example, the focus should be on die extent to which the influences of 
Head Start, school and die family combine to provide die child with opportunities to learn 
and function in an educational setting at an optimal level consistent widi his or her ability. 

The overall Head Start evaluation agenda should strive to balance die fol'owing elements- 
validity of outcome measures, ease of data collection for bodi researchers and program 
staff, muldmediod assessment strategies, and an understanding of the underlying dynai acs 
diat uie indicators purport to represent 

6. Data collection procedures and techniques must be valid and appropriate 
for the particular research question and the specific population. 

Any norm-referenced instniments used vith particular subpopulations must be valid for the 
suopopulation under consideration. Interviews and questionnaires administered to parents 
must take :nto account literacy level and linguistic usage of die respondents. In order to 
insure reliability and validity of the responses, diese instruments may require special 
adaptanons for different subpopulations. Of particular importance is attention to predictive 
validity. Reliable instruments may differ substantia"y in their capacity :o predict the same 
outcome for different subpopulations. Therefore, most instniments will, of necessity have 
to be pilot-tested prior to their use in research and evaluation studies. 

AH instruments should be administered individually to Head Start children. To optimize 
measureinent comparability across years, individually administered tests also should be 
employed in later grade school followups. 



7* Program variation must be explored while searching for explanations of 
differential outcomes. 

There is no single active ingrr diem in Head Start that is the key to program outcomes. 
Local Head Start programs arc complex organizational entities that interact with equally 
complex fanuly and community ecologies. This calls foe a ccxnbination of holistic research 
strategies (includfing wganizational, case study, and ethnographic niethodologies) together 
with multivariate statistical explorations involving natural variation, experimental, and 
quasi-cxpeiimcntal designs. Researchers should be alert to the possibilities that interaction 
of program attributes may be more important in accounting for outcomes than individual 
program attributes considered in isolation. 

The following program variables arc aii ig those that are particularly important: 

staff characteristics and behaviors (including education, years of experience, 
credentials and certification, length and type of training, and knowledge of 
early childhood education and chUd development); 

classroom composition (group size and child- staff ratio) and, for home- 
based programs, family-home visitor case loads; 

nature and intensity of interactions among children and between children 
and staff; 

curricular strategy, including the extent to which the curriculum is well- 
planned anc appropriately delivered; 

nature and intensity of parent involvement; 

staflf-parcnt interaction; 

nature and frequency of home visiting and other family contacts; 

delivery of comprehensive services, inclr.ding nature of linkages with 
health, nutrition, social and educational agencies; 

length and intensity of child*s participation per day, week, and year, 

age at onset of intervention and number of years of child*s and family's 
participation in the program; 

degree of program responsiveness to identified needs of participating 
chUdien and families; 

extent and nature of flexibility exercised by the local program in tailoring its 
delivery system to specific circumstances and resources in the community; 

organizadonal climate; 

administrative and personnel issues (including staff compensation and 
turnover); and 
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program auspices. 



SiS^ should capitabze on the insights of prioi and ongoing research in early 

deve opitient. and other related fields in exploring the e 
vanabl^. For example, organizational research is cunenUy bringing new understiidings 
L^^^c",?°"'"^- S^"^^" perspectiies on examining S sSt 

' • P™^^' P^^^'f"^ tools. Ar4s of inquiry 

mcludc leadership, how decisions are mads, the role of parent participation on Dolicv 

^^8^^ relationships of lockl pro^ a^3 ^CYF 
approaches to program implementation. Related aLinistrative aii 
f "^T^^ °^ P"*f ^ attention, as suggested by recent findings in child 
r!!^r!^ '^"8 relationship between staff salaries and tumovefand child 

T„H°J^-^^°" programs, variation within programs must be explored. 
nToSS. 1 ^^k"^ ^°1??^J:!^^ly ^" the attributes with which they enter Head Stan 
programs. These atmbutes will diffei in theii interactions with program characteristics 
^^"Sc'T ^^'1 with children in the same classrooms will va? in their Head btan 
t^T^^' ^ u^l? methodology assumes a uniform treatment across families, at least 
n^^'ffrS ' ^ ^l^oom. In studies which explore the reasons for 

S ofAeSy. ^° incorporate consideration of such variation in the initial 

SSi^i^*"* r^earch and evaluation studies can be greatly enhanced by 
buildmg on the existmg strengths of programs and program staffs. 

W^nSSi\'t^"?i^ researchers need to approach all studies with a focus on 
i7f?H;wfnr ^^^^ °^ '^f'^^u P'°S^ P^°S^ staffs, rather than emphasizing 
deficit (for similar reasons that the approach to children and families -chould emphasize 
fi^fi than deficits). Research projects are most likely to succeed if researchers 

S,h...c^i ? °^ P'°j^t, including the initial development of 

n^n?nl^',h^S°H r°" S^T^" ^ ! '^'^^^^ P"»8™" communities calls for joint 
planning with clearly defined roles and benefits for all parties. 

ficelSnce hnfh'?? P'^^^' "'"^ ' ' ^hey involve a search for 

excellence both m overall programs and . ;;^icular strengths within programs In 

n^^Sfr^'-Sr"^""'^^ ^""''T !PJ"^ P^^S^^'" f^^tures will be most successfuT^ tho e 
l^rSTr. ^ f considered by Head .Stan staff as new and exciting additions to their 

uSSentinTthe^i^^^^^^^ ^° '"^^^^^^ ^""^^^ ^° ^^-^ ^^e costs of 

Although cooperative efforts between program and evaluation staff are essential, data 
collection procedures must include safeguards to insure that findings are not subject to the 
°f ^"P°"^«"^ I'i^s- Such safeguards might include the use of rftings by 
independent observers not connected with a program. In addition, program staff should 
^oSnfT^'" findings will not oe used by adminisKs S pSal ze Ae 



Budgetary Considerations 



The Panel's chief concern is that high quality research is conducted and used to make future 
decisions about programs that can profoundly affect the lives of poor children and their 
families. In addition to the reconmiendcd principles that undergird the research strategy 
^iscusf^ed above, the following corollary principles involve budgetary consideraticms. 

• High quality research is expensive. A smaller number of adequately funded 
studies is preferred to a larger number of inadequately funded efforts. 

• The first year of all large scale and complex studies should be devoted to 
design and pilot tesdng. For all studies, there should be sufficient lead time 
priOT to the formal data collection stag, to permit careful pilot testing and the 
refining and Hne-tuning of the design, measurement instruments and 
procedures. 



ERIC 



-12-14 



RECOMMENDED RESEARCH DIRECTIONS 



Thus far, the rccommendatioKS in this report have centered on the overall strategy and 
design issues embodied in a program of research and evaluatioij to address the questions: 
*What woits best for whom?" and "What effects are maintained?" In keeping with this 
thorust, the Panel leconmicnds a series of research directions that are important for building 
a future knowledge base. The specific studies that evolve from these recci imcndations 
should reflect both ACYFs priorities and the important research issues identified by the 
larger research community. 

No distinction is made between evaluation and research with respect to these strategic 
design considerations. The common theme of the research directions set forth below is a 
series of studies to explore efficacy questions and, more broadly, to produce information to 
fill existing gaps in knowledge. The general thrust is toward models that help us to 
interpret dtffercnti»J outcomes and that should be incorporated into efficacy studies, 
whetfier termed research or evaluation projects (not a terribly useful distinction for these 
purposes). 

The Panel recommends tiiese four research directions for inclusion in the overall strategy: 

I. Longitudinal studies that seek to identify early and intermediate 
outcomes of a Head Start experience and that explore the interacting 
influences of preschool, family, and later schooling in mediating the 
long-term effects of child and family participation in Head Start 

Head Start research should address the total flow of child and family outcomes during and 
subsequent to participation in tne preschool program* In particular, the Panel urges placing 
a priority focus on the trajectory of the child and family outcomes following the Head Start 
experience. This would include short-term, medium-term, and, whenever feasible, long- 
term gains (and losses) attributable to Head Start and to the subsequent stream of influences 
associated with the family, public schools, and other relevant institutions that have 
significant effects on the child-family ecology. 

Since the beginning of Head Start, many have been concerned that the experiences of Head 
Start graduates in die public schools migbt vitiate positive bcntilts of the program. Early 
findings of Head Start evaluations showed a regular pattern of decline iri the first few years 
of school suggesting that the benefits of Head Start— as indexed by standardized school 
achievement tests — were largely gone by the third or fourth grade. Findings ft^om the 
Perry Preschool Project and the larger group of studies included in the Consortium for 
Longitudinal Studies of graduates of university based intervention programs have shown a 
latent longcr-tenn effect of children's experiences in the earlier years on a variety of other 
outcomes (for example, dropout rates, placements in special classes, and retention in 
grade). These results suggest that the early intervention may have had a significant effect 
on variables associated with children's later performance. 

Long-term outcomes of social and economic importance produced by preschool programs 
is a pivotal issue that is likely to affect the Head Start policy climate for many years to 
come. The Panel is persuaded that the lasting benefits identified in some research-oriented 
preschool projects are likely to be genuine and not an artifact. How /er, policymakers and 
the general public should not be oversold that early education and intervention programs 



ERiC 



-13- 



such as Held Stan, even when implemented in a high quality fashion, are some kind of 
panacea that succeed even in the absence of appropriate ongoing child and fa >ily support. 

Little is known about the specific factors in the preschool program, within the family, in 
later school experiences, and in the total child and family ecology that sustain, reinforce and 
extend these benefits. Research that focuses on near-term and medium-term outcr-nes (at 
least through fourth, fifth, or Pixth grade) pait of :j\ effort to identify those progiixm and 
family processes that arc associated with lasting gains is particularly worthy of ACYF 
support. Such research shcyld include short-term outcome measures that^ attempt to 
anticipate long-term effects and mid-tenn antecedents of such progress. Such studies of 
explanatcwy models should shed light on the r^.ystery of long-term outcomes of high quality 
preschool programs. If feasible, ivjsearch should be designed so that the option is available 
to extend at least a portion of th: studies for a longer period, potentially until adulthood. 

These studies also might include Head Stan eligible children in other preschool settinf^s or 
use an epidemiological approach which would identify a large low income population 
group at age 2 or S and longimdinally follow these children and families whether or not 
they have Head Start or other preschool experiences. An advantage of an epidemiological 
model is the possibility of picking up children with no type of formal prescnooi or child 
caie experience. Foundations and other organizations, including the Department of 
Education and state and local education agencies, could be inx'^ted to collaborate in a series 
of longiti^dinr^ * search subsidies targeted on this issue. 

In addition i > this major recommendation for a longitudinal exploration of explanatory 
models of lasting effects, other studies proposed below may include longitudinal aspects 
designed to accomplish other purposes. 

2. Studies Co identify quality ingredients in existing Head Start programs. 

The variations within the Head Stan program include issues of quality. Because Head Start 
provides such a naUiral laboratory, it is possible to explore issues of quality for either total 
programs or for particular components or variables in terms of "What works best for 
whom?'' Case, ethnographic, and correlational studies would be particularly useful for 
fomaulating hypotheses about critical variables under different conditions. Second stage 
studies either could use experimental methods to test these hypotheses or study the 
prevalence of quality ingredients across Head Start programs. 

A research area of particular importance is studies of parent involvement and family 
support. ACYF is giving increasing attention to these issues and this program fv>:us on 
strategies to promote parent participation and strengthen families is the topic of a gt owing 
number of books and articles on low income and minority families. However, the e has 
never been a major research or evaluation study solely focused on parent involvemer t and 
family support in He \d Start. The Panel strongly believes that these topics wi^rrant 
considerable research emphasis. Such a program of studies should take into account the 
types of parent involvement, the complexity of alternative family structures, percei ^ed 
and/or actual family interaction processes, and sociocultural and language characteristics of 
subgroups of families. As in the case of other major research emphases described above, 
these studies can be pursued most fruitfully through a multimethod approach, involving a 
consortium of researchers. 



ERLC 



-14- 



In the early studies. Head Start should examine its administrative data bases for their 
usefuln'^ss in identifying either quality programs or programs that have high quality 
examples of the component or variable under consideration. 

3 • Studies of Head Start's emerging innovative pro^^,ram strategies. 

The first two recommended research directions focused on the mainstream Head Start 
program as it has evolved over the past 25 years. ACYF is currently initiating a number of 
innovative program strategies. Some of tfiese may begin as small scale demonstrations 
while others may be phased in quickly and incorporated in the nationwide effort. At the 
start of innovative projects, the demonstration and the evaluation should be planned 
siuiultaneously and interactively. Candidates for evaluation initiatives include promising 
Head Start program innovations such as family literacy, transition, and job training through 
the Family Support Act Continuity between very early intervention programs and Head 
Start is another important research topic (for example, the Comprehensive Child 
Development Program, Paren' Child Centers, and P, L. 99-457 activities for infants and 
toddlers with disabilities migh ^ candidates for longitudinal inquiry). 

4 • Studies of special subpopulations of Head Start and other priority 

research and evaJjation issues. 

Special studies should target Head Start subpopulations that may not be included in 
sufficient numbers in research and evaluation studies recommended above. For 
example, Hispanics, Native Americans, migrant farmworker families, Asians, children 
with disabilities, and geographically and socially isolated families have frequendy been left 
out and have rarely been the principal focus of Head Start or other early childhood research 
and evaluation projects, TliC emphasis on subpopulations is intended to place greater 
priority on the inclusion of a wide diversity of subgroups in all studies; however, it may 
not always be feasible to include sufficient numbers of participants from small subgroups 
in studies that are intended co focus on other purposes. 

Certain studies should be des^^ned to focus specifically on the special needs and 
characteristics of very small Hcaa Start subpopulations. Such studies should identify those 
research themes that are common to other Head Start populations and those that are specific 
to the particular subpopulation. In addition, these studies can identify exemplary program 
strategies that are responsive to the needs of particular groups. 

This effort should include studies of special topics that may not be adequately dealt with in 
the studies identified above. Candidates for such special sar'es are evaluations of health 
services (including medical, denial, mental health, nutritioh, nd of special services to 
children with disabilities. 

Another type of special study is an analysis of the Head Start intervention itself as an 
outcome variable. How have local Head Start programs tailored their intervention to 
respond to the special needs of groups in their conmiunity and acted to mobilize available 
community resources? The local grante^delegate agency*s approach to implementing the 
program performance standards would be viewed as a 'Tu-st level outcome" worthy of 
exploiation in its own right. The objective would be to explore the practical operational 
wisdom of local program administrators in using the flexibility that Head Start allows and 
encourages. 
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RECOMMENDATIONS FOR RESEARCH SUPPORT ACTIVITIES 



The above two sections of this paper set forth general principles and a research strategy for 
Head Start inquiry and recommend four specific sets of studies for ACYF consideration. 
In order for this strategy and these recommendations to achieve the desired results over 
time—improving Head Start quality — it is essential to create an environment which 
nurtures research and evaluation. The following four research support activities are 
proposed to meet this need 

1 . Establish an archive of signiflcant Head Start data. 

To be truly scientific, research must be cumulative in nature. The era of the one shot 
megastudy is over, or soon should be. A unifying theme of the Paners recommendations 
is the impOTtance of ACYF speariieading an interrelated scries of studies in partnership with 
the larger research community. In order for this strategy to succeed, it is imperative to take 
steps to gain easy access to research findings. Related to this is the need to be able to 
access research data bases in electronic form in order i) conduct secondary analyses and for 
other purposes. 

The Panel rccommer Is, therefore, that ACYF establish a Head Start data archive of 
significant studies, aJoL ^; with procedures for qualified researchers conveniently to access 
an electronic data flle nth appropriate confidentiality safeguards. The archive should 
include all Head Stan studies sponsored by ACYF and by oAer funding sources. ACYF 
should specify archiving instructions and requirements at the time of funding Head Start 
research. Some additional resources may be necessary to acquire data from otiier research 
sponsors and to put them in a convenient format for ready access. If resources permit, 
funds mi^t be made available to facilitate use of the archive by any investigator. The costs 
of the HMd Start research archive would be modest; the benefits would be considerable. 

As a first step toward meeting this objective, ACYF might consider establishing a work 
group or research council to discuss data archiving as part of the broader process of 
research planning and dissemination. 

2 • Develop a plan for Head Start measures identification and development. 

As noted above, outcome measures must be identified that capture salient child and family 
outcomes and, where appropriate, community and institutional outcomes. AC\T should 
take the lead in the effort to identify marker variables and appropriate outcomes. Where 
child and family outcome measures do not exist in usable forms, initiatives should be 
considered to modify existing instruments or to develop new measures. In general, 
because the development of high quality measures is typically a complex, time-consuming, 
and costly undcrtaJdng, a new measures development effort should be undertaken only 
when a thorough review of existing instruments indicates a real need for developing a new 
measure, fox ♦he purpose at hand. 

There is a need for measures that focus on child social competence and family functioning 
in a two generation approach. The emphasis should be on relatively straightforward 
indicators of the child's developmental profile (including cognitive and language abilities, 
social functioning relevant to later success in school, and physical development), the 
family's vulnerability/strength profile, and parent-<:hild interaction, as discussed earlier in 
this report. Measures might address questions such as: What are valid indices of the 
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'° u5 "u^^ factors that affect low income and minority children and families? 
Multiniethod observational strategies are particularly promising. 

As a first step, ACYF could hosi small conferences bringing together early intervention 
researchers who have develope^: or adapted measures with high oromise of utility in Head 
Start research. Bringing together experienced researchers would be a way of sharine 
information about state-of-the-art measures. In view of the criticality of long-tenS 
outcomes, representauves of the major research-oriented preschool projects that have 
Identified lasting effects should be included in the planning. Similarly, special efforts 
should be made to take into account issues related to the longitudinal measurement of 
chan^ in child outcomes m variou- domains over time. Since it is expected that measures 
idennficauon and development will involve extensive coUaboration between researchers and 
Head Start programs, representatives of local programs also should be included in the 
conferences. 

ACYF can ask researchers to draft monographs focused on particular outcome domains 
either pnor to or following these planning conferences. The following are examples of 
some of the acuviues that may flow from the measures conferences: 

• Funding by ACYF of some projects to develop new measures; 

• Implementation of projects to refine measures that show special promise 
(for example, adaptation of straightforward measures, measures for 
different ethnic groups, measures for children of different ages and 
developmental levels); 

• Identification and/or development of a core set of marker variables that 
could be used in common across many Head Stan research and evaluation 
studies in order to gather comparable data and to support meta-analysis and 
other forms of secondary analysis; 

• Identification of computer software packages that facilitate the analysis of 
complex areas, such as longitudinal measurement cf change; 

• Establishment of panels to advise on future measure;; activities; and 

• Establishment of a clearinghouse on early intervention measures. 

3. Develop and implement a strategy for Head Start research and evaluation 
capacity building. 

Capacity building is necessary to establish the institutioP-l mechanisms and collaboration 
between Head Start and the overall research community that arc essential to carrying out the 
strategy ouUined above. Capacity building refers to creation of the infrastructure necessary 
to implement the research strategy proposed by the Panel. Some capacity building can be 
accomplished within the framework of specific studies (for example, using the consortium 
technique of conducting several small substudies within the umbreUa of one larger project) 
Another approach is through joint or parallel funding with other sources (for example! 
foundations or research institutes). Additional capacity building efforts should be 
considered by ACYF beyond what occurs in individual research projects. 
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ACYF might establish an ongoing institutional process or processes to link in-house 
research and evaluation planning and management to the broader research community. 
Possible steps could be to host workshops bringing together researchers and early 
intervention program specialists to consider the latest thin^ng and research tools and to 
commission monographs dealing with critical issues of research and evaluation 
methodology. Other steps could include assistance in obtaining software programs for data 
analysis. 

Monographs should include appropriate uses of experimental and quasi-experimental 
methods, multimethod observational techniques, and other approaches. The monographs 
should identify the types of studies and issues for which particular methodologies are 
suited and the pros and cons of each approach in terms of criteria such as clarity of causal 
inference, cost, timing, practicability, burden on grantees and families, and ethical 
considerations. 

One mechanism for developing monographs would be to support graduate students in their 
studies for work on priority issues identified by ACYF, Such a plan is already being 
considered within the agency. Monographs that have greater time urgency or that require 
particular expertise could be commissioned from established researchers. 

4. Utilize information from existing Head Start administrative data bases 
for research and evaluation purposes. 

In addition to their utility for program management, monitoring, and other operational 
purposes, Head Start administrative data bases can play useful roles in research and 
evaluation. Existing Head Start administrative data bases such as the Program Information 
Report (PIR) and the Head Start Cost System (HSCOST), and the forthcoming child and 
family data base should be incorporated in the data archive. PIR data can prove useful in 
drawing samples of grantees and delegate agencies for research studies. These data may 
also serve as proxy indicators for program outcomes or for program processes and 
variables believed to be associated with child and family outcomes. As progress is made 
toward identifying a set of marker variables, these variables should be reviewed and 
incorporated in administrative data bases whenever feasible. 



CONCLUSION 



In its 25 years of existence, Head Start has faced increasingly complex issues in meeting 
the needs of low income children and families. Although the importance of Head Start is 
well recognized, providing services that meet the needs of increasingly diverse populations 
will continue to be a major challenge. There is a need for reliable and valid information 
about what worics, why it worics, for whom it works and under what conditions it works. 
The answers to these questions, based on the findings of quality research and evaluation 
studies, can be a potent force in preserving and promoting the well-being of this country's 
most important resource, its children. 
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